> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Anny26022/chartsmaze_clone/llms.txt
> Use this file to discover all available pages before exploring further.

# fetch_bulk_block_deals.py

> Fetches bulk and block deal transactions from the last 30 days using Dhan Static ScanX API

## Overview

Fetches bulk and block deal data for the last 30 days from the Dhan Static ScanX API. Due to API limitations (max 10-day date range per request), the script fetches data in three 10-day chunks and deduplicates results.

**Source:** `fetch_bulk_block_deals.py`\
**Phase:** Phase 2 (Enrichment)\
**Output:** `bulk_block_deals.json`

## API Endpoint

```
POST https://ow-static-scanx.dhan.co/staticscanx/deal
```

## API Limitations

<Warning>
  The API enforces a maximum date range of **240 hours (10 days)** between start and end dates. Requests exceeding this will fail.
</Warning>

## Request Payload

<ParamField path="data.startdate" type="string" required>
  Start date in DD-MM-YYYY format
</ParamField>

<ParamField path="data.enddate" type="string" required>
  End date in DD-MM-YYYY format (max 10 days from startdate)
</ParamField>

<ParamField path="data.defaultpage" type="string" default="N">
  Whether to use default pagination
</ParamField>

<ParamField path="data.pageno" type="int" required>
  Page number (starts at 1)
</ParamField>

<ParamField path="data.pagecount" type="int" default="50">
  Number of results per page
</ParamField>

### Example Request

```json theme={null}
{
  "data": {
    "startdate": "01-01-2024",
    "enddate": "10-01-2024",
    "defaultpage": "N",
    "pageno": 1,
    "pagecount": 50
  }
}
```

## Function Signature

```python theme={null}
def fetch_bulk_block_deals():
    """
    Fetches bulk/block deals for the last 30 days in 3 chunks of 10 days each.
    Auto-paginates through all pages for each chunk.
    Deduplicates results and saves to bulk_block_deals.json.
    """
```

## Pagination Logic

The script auto-paginates based on `totalcount` from API response:

```python theme={null}
page_no = 1
max_pages = 1  # Will be updated from first response

while page_no <= max_pages:
    # ... make request ...
    
    total_count = data.get('totalcount', 0)
    if total_count > 0:
        max_pages = math.ceil(total_count / 50)
    
    page_no += 1
```

## Date Chunking Strategy

Fetches 30 days in three 10-day chunks:

```python theme={null}
for i in range(3):
    days_offset_end = i * 10 
    days_offset_start = days_offset_end + 9
    
    chunk_end_date = end_date_ref - timedelta(days=days_offset_end)
    chunk_start_date = end_date_ref - timedelta(days=days_offset_start)
    
    start_str = chunk_start_date.strftime("%d-%m-%Y")
    end_str = chunk_end_date.strftime("%d-%m-%Y")
```

**Example:**

* Chunk 1: Days 0-9 ago (most recent)
* Chunk 2: Days 10-19 ago
* Chunk 3: Days 20-29 ago

## Deduplication

Results are deduplicated using a composite key to handle overlapping chunks:

```python theme={null}
unique_deals_map = {}
for d in all_raw_deals:
    key = f"{d.get('sym')}_{d.get('date')}_{d.get('qty')}_{d.get('avgprice')}_{d.get('bs')}_{d.get('cname')}"
    unique_deals_map[key] = d

sorted_deals = sorted(list(unique_deals_map.values()), key=lambda x: x.get('date', ''), reverse=True)
```

## Output Structure

<ResponseField name="sym" type="string">
  Stock trading symbol
</ResponseField>

<ResponseField name="date" type="string">
  Deal date
</ResponseField>

<ResponseField name="qty" type="number">
  Quantity traded
</ResponseField>

<ResponseField name="avgprice" type="number">
  Average deal price
</ResponseField>

<ResponseField name="bs" type="string">
  Buy (B) or Sell (S) indicator
</ResponseField>

<ResponseField name="cname" type="string">
  Client/counterparty name
</ResponseField>

### Additional Fields

The API response may include additional fields like:

* Deal type (Bulk/Block)
* Exchange
* Remarks

### Example Output

```json theme={null}
[
  {
    "sym": "RELIANCE",
    "date": "2024-01-15",
    "qty": 5000000,
    "avgprice": 2450.50,
    "bs": "B",
    "cname": "ABC INVESTMENTS LTD"
  }
]
```

## Dependencies

* `requests` — HTTP client
* `json` — JSON parsing
* `datetime` — Date range calculations
* `math` — Pagination math (ceil)
* `pipeline_utils.py` — Provides `get_headers()` function

## Configuration

Hardcoded configuration:

```python theme={null}
url = "https://ow-static-scanx.dhan.co/staticscanx/deal"
headers = get_headers()
end_date_ref = datetime.now()  # Today
page_size = 50
chunks = 3  # 3 x 10-day chunks = 30 days
```

## Error Handling

* 10-second timeout per request
* Breaks pagination loop on HTTP errors or empty responses
* Continues to next chunk if one chunk fails
* Prints detailed progress and error messages

```python theme={null}
try:
    response = requests.post(url, json=payload, headers=headers, timeout=10)
    if response.status_code == 200:
        # Process data
    else:
        print(f"  Error fetching page {page_no}: Status {response.status_code}")
        break
except Exception as e:
    print(f"  Exception fetching page {page_no}: {e}")
    break
```

## Progress Tracking

```python theme={null}
print(f"Fetching deals for chunk {i+1}/3: {start_str} to {end_str}...")
print(f"  Fetched page {page_no}/{max_pages} ({len(deals)} items)")
```

## Usage Example

```bash theme={null}
python3 fetch_bulk_block_deals.py
```

**Expected Output:**

```
Fetching deals for chunk 1/3: 24-02-2024 to 03-03-2024...
  Fetched page 1/3 (50 items)
  Fetched page 2/3 (50 items)
  Fetched page 3/3 (25 items)
Fetching deals for chunk 2/3: 14-02-2024 to 23-02-2024...
  Fetched page 1/2 (50 items)
  Fetched page 2/2 (30 items)
Fetching deals for chunk 3/3: 04-02-2024 to 13-02-2024...
  Fetched page 1/1 (42 items)
Successfully saved 247 unique bulk/block deals to bulk_block_deals.json
```

## Integration

This script is part of **Phase 2 (Enrichment)** in the EDL Pipeline. The output file is consumed by:

* `add_corporate_events.py` — Adds "📦: Block Deal" event markers for deals in last 7 days

Run via master pipeline:

```bash theme={null}
python3 run_full_pipeline.py
```

## Performance Notes

* Total API calls: \~3-10 requests (depending on deal volume)
* Typical runtime: 5-10 seconds
* No threading (sequential pagination required)
* Data volume: Typically 200-500 deals per month